1 1. A machine-based method for use in processing a 

2 digital image that includes first and second regions, the 

3 method comprising: 

4 estimating an intrinsic color of a given pixel 

5 located in an area of interest that is adjacent to at least 

6 one of the first and second regions, the estimating 

7 comprises extrapolating from colors of multiple pixels in 

8 one of the first and second regions and multiple pixels in 

9 the other of the two regions, and 

10 storing the intrinsic color of the pixel for later 

11 use. 

1 2 . The method of claim 1 in which the given pixel 

2 has an original color that relates to the original colors of 

3 pixels in both the first and second regions, and the 

4 estimated intrinsic color of the given pixel relates to 

5 original colors in only one or the other of the first and 

6 second regions . 



1 3. The method of claim! in which the area of 

2 interest includes one of the first and second regions. 

1 4 . The method of claim 1 in which the area of 

2 interest is adjacent to both of the first and second 

3 regions . 

1 5. The method of claim 1 in which the first region 

2 comprises a foreground object and the second region 

3 comprises a background. 
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1 6. The method of claim 1 in which the first and 

2 second regions may have any arbitrary degree of color 

3 variation in the visible spectrum over a spatial scale that 

4 is on the same order of magnitude or smaller than the 

5 minimum span of the area of interest. 

1 7. The method of claim 1 in which the estimating 

2 comprises analyzing both the color and spatial proximity of 

3 pixels in the first and second regions. 

1 8. The method of claim 1 in which the estimating 

2 comprises extrapolating from the closest pixels in the first 

3 and second regions. 

1 9 . The method of claim 1 in which the estimating 

2 comprises flowing colors into the area of interest from one 

3 or both of the first and second regions. 

1 10. The method of claim 9 in which the flowing of 

2 colors comprises averaging of color values for each of a set 

3 of pixels in the first region and a set of pixels in the 

4 second region. 

1 11. The method of claim 1 in which the digital 

2 image comprises layers of pixel information and the 

3 estimating is based on pixel information in only one of the 

4 layers . 

1 12. The method of claim 1 in which the digital 

2 image comprises layers of pixel information and the 

3 estimating is based on pixel information in a composition of 

4 all the layers. 
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1 13. The method of claim 1 further comprising 

2 determining an opacity value for the given pixel, indicative 

3 of the extent to which the intrinsic color of the given 

4 pixel relates to original colors in the first and second 

5 regions, based on a result of the estimating of the 

6 intrinsic color. 



1 14. The method of claim 13 in which the given pixel 

2 includes original opacity information, and the opacity value 

3 is also based on the original opacity information. 

1 15. The method of claim 13 further comprising 

2 determining opacity values for other pixels that have 

3 intrinsic colors that relate to original colors in the first 

4 and second regions. 

1 16 . The method of claim 13 in which the opacity 

2 determination comprises use of a neural network trained on 

3 the image original colors and estimated intrinsic colors. 

1 17. The method of claim 13 further comprising using 

2 the opacity value to composite one of the first and second 

3 regions with another digital image. 

1 18. The method of claim 1 in which the estimating 

2 also includes extrapolating estimates of intrinsic colors of 

3 the first and second regions using searches in color space 

4 and image coordinate space. 

1 19. The method of claim 1 in which the estimating 

2 assumes a linear blending model. 
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1 20. The method of claim 1 in which the estimating 

2 includes flowing colors from edges of the area of interest 

3 to fill the area of interest with estimates of the colors of 

4 the first and second regions. 

1 21. The method of claim 1 further comprising 

2 extracting from the digital image the intrinsic 

3 colors of the given pixel and of other pixels that have 

4 intrinsic colors that relate to original colors in the first 

5 region or second region. 

1 22 • The method of claim 21 further comprising using 

2 the extracted intrinsic colors to composite the first region 

3 or the second region with another digital image. 

1 23. The method of claim 1 further comprising 

2 receiving from an interactive user interface 

3 information that identifies the area of interest. 

1 24. The method of claim 1 in which estimating the 

2 intrinsic color comprises 

3 determining two color sample sets for the given 

4 pixel, each of the color sample sets being associated with 

5 one of the first and second regions, and 

6 estimating the intrinsic color based on the two 

7 color sample sets. 

1 25. The method of claim 24 in which estimating the 

2 intrinsic color comprises comparing the original color of 

3 the given pixel with colors in the color sample sets. 
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1 26. The method of claim 24 further comprising 

2 determining an opacity for the given pixel 

3 indicative of the extent to which the intrinsic color of the 

4 given pixel relates to original colors in both of the first 

5 and second regions, where the determination of opacity 

6 includes comparing the original color of the given pixel 

7 with colors in the color sample sets. 

1 27. The method of claim 26 in which the given pixel 

2 includes original opacity information and the determination 

3 of opacity is also based on the original opacity 

4 information. 

1 28. The method of claim 24 in which the color 

2 sample sets are derived from colors of pixels in the first 

3 and second regions . 



1 29. The method of claim 24 in which a single color 

2 is selected from each of the color sample sets based on an 

3 error minimization technique. 

1 30. The method of claim 1 in which the intrinsic 

2 colors of all of the pixels in the area of interest are 

3 determined automatically. 
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1 31. A machine -based method for use in processing a 

2 digital image, the method comprising: 

3 enabling a user to paint an area of the digital 

4 image to identify at least an area of interest adjacent to 

5 at least one of a first region and a second region, 

6 after the user has defined the area of interest, 

7 automatically estimating intrinsic colors of pixels in the 

8 area of interest based on color information for pixels in 

9 the first region and the second region. 

1 32. The method of claim 31 in which the painting is 

2 done with a brush tool that can be configured by the user. 

1 33. The method of claim 31 in which the painted 

2 area can be built up by repeated painting steps and in which 

3 portions of the painted area can be erased by the user 

4 interactively. 

1 34. The method of claim 31 in which the user paints 

2 the area of interest and separately identifies a location 

3 which is in one of the first and second regions. 

1 35. The method of claim 31 in which the user paints 

2 at least one of the first and second regions and the area of 

3 interest and separately identifies a color associated with 

4 one of the first and second regions. 

1 36. The method of claim 31 in which the user 

2 designates one of the first and second regions by 

3 identifying a pixel location in that region. 
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37. The method of claim 35 in which the user 
identifies the color by applying an eyedropper tool to one 
pixel or a set of pixels in the one region. 

38. The method of claim 31 further comprising flood 
filling one of the regions based on the identified pixel 
location to designate that region as a foreground. 

39. The method of claim 31 in which one of the 
first and second regions comprises a foreground in the 
digital image, the other of the regions comprises a 
background of the digital image, and the area of interest is 
between the foreground and the background. 

40. The method of claim 31 in which one of the 
first and second regions comprises a foreground in the 
digital image, the other region comprises a background of 
the digital image, and the area of interest is adjacent to 
one of the regions and includes at least part of the other 
region. 

41 The method of claim 31 in which the painted 
area may be modified by a user interactively and repeatedly. 

42 The method of claim 31 in which the user is 
enabled to paint additional areas of interest between other 
pairs of first and second regions. 
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1 43. A method for use in processing a digital image, 

2 comprising 

3 receiving a mask associated with an area of interest 

4 in the digital image, the mask including values representing 

5 opacities of pixels in the region of interest with respect 

6 to an adjacent region of interest, and 

7 based on the mask, estimating intrinsic colors for 

8 the pixels. 

1 44. A machine-based method for use in extracting a 

2 foreground region from a background region of an image, 

3 comprising 

4 enabling a user to control an original extraction by X 

5 manipulating a brush on a display of the image, 

6 enabling a user to control a touch up extraction 

7 following the original extraction, and 

8 considering a pixel identified for touch up 

9 extraction only if the pixel was of uncertain color in the 

10 original extraction. 

1 45. The method of claim 44 in which an intrinsic 

2 color is determined for each of the pixels that were of 

3 uncertain color based on a forced foreground or background 

4 color. 

1 46. The method of claim 44 in which the forced 

2 color is selected by the user. 

1 47. The method of claim 44 in which the forced 

2 color is determined automatically from the original colors 

3 within the foreground region. 
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1 48, A method for use in determining, for each pixel 

2 in an area of interest in a digital image, the nearest pixel 

3 in a first region of the image that is adjacent to the area 

4 of interest and the nearest pixel in a second region of the 

5 image that is adjacent to the area of interest, the method 

6 comprising 

7 defining a processing area that is smaller than the 

8 image , 

9 defining a pixel window that is smaller than the 

10 defined processing area, 

11 scanning the processing area to a succession of 

12 overlapping positions that together span the image 

13 at each overlapping position of the processing area, 

14 scanning the pixel window across the processing area, and 

15 at each position of scanning of the pixel window, 

16 updating stored information for pixels in the window, the 

17 stored information relating to nearest pixels in the first 

18 and second regions . 

1 49. The method of claim 48 in which the processing 

2 area comprises a rectangle twice as long is high, and in 

3 each of the succession of positions the processing area is 

4 offset from the prior position by half the length of the 

5 rectangle. 

1 50. The method of claim 48 in which the pixel 

2 window comprises a square. 

1 51. The method of claim 48 in which the scanning of 

2 the processing area and the scanning of the pixel window 

3 occur in both forward and backward passes that span the 

4 image . 
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1 52. The method of claim 48 further comprising 

2 extrapolating colors from the nearest pixels. 

1 53. The method of claim 48 in which the first 

2 region comprises a foreground object, the second region 

3 comprises a background, and at least some pixels in the area 

4 of interest have uncertain color. 

1 54. A machine-based method for a user to extract an 

2 object from a background in an image, comprising 

3 displaying the image, 

4 selecting a painting tool and adjusting its 

5 characteristics , 

6 using the painting tool to paint a swath around the 

7 object, 

8 the swath including pixels whose membership in the 

9 object or the background are uncertain and including pixels 

10 that with certainty belong to the object and to the 

11 background 

12- indicating at least one pixel that is known to 

13 belong to the object or the background, 

14 invoking a program to perform the extraction, 

15 observing whether the quality of the extraction, and 

16 depending on the observation, using a painting tool 

17 to control a touch-up extraction. 
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1 55. A medium bearing a computer program capable of 

2 controlling a computer to process a digital image that 

3 includes first and second regions by: 

4 estimating an intrinsic color of a given pixel 



5 located in an area of interest that is adjacent to at least 

6 one of the first and second regions, the estimating 

7 comprises extrapolating from colors of multiple pixels in 

8 one of the first and second regions and multiple pixels in 

9 the other of the two regions, and 



10 storing the intrinsic color of the pixel for later 

11 use. 

1 56 . A medium bearing a computer program capable of 

2 controlling a computer to process a digital image by: 

3 enabling a user to paint an area of the digital 

4 image to identify at least an area of interest adjacent to 

5 at least one of a first region and a second region, 

6 after the user has defined the area of interest, 

7 automatically estimating intrinsic colors of pixels in the 

8 area of interest based on color information for pixels in 

9 the first region and the second region. 

1 57. A medium bearing a computer program capable of 

2 controlling a computer to extract a foreground region from a 

3 background region of an image by 

4 enabling a user to control an original extraction by 

5 manipulating a brush on a display of the image, 

6 enabling a user to control a touch up extraction 

7 following the original extraction, and 

8 considering a pixel identified for touch up 

9 extraction only if the pixel was of uncertain color in the 

10 original extraction. 
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1 58. A system for use in processing a digital image 

2 that includes first and second regions, the system 

3 comprising: 

4 means for estimating an intrinsic color of a given 

5 pixel located in an area of interest that is adjacent to at 

6 least one of the first and second regions, the estimating 

7 comprises extrapolating from colors of multiple pixels in 

8 one of the first and second regions and multiple pixels in 

9 the other of the two regions, and 

10 means for storing the intrinsic color of the pixel 

11 for later use . 

1 59. A system for use in processing a digital image, 

2 the system comprising: 

3 means for enabling a user to paint an area of the 

4 digital image to identify at least an area of interest 

5 adjacent to at least one of a first region and a second 

6 region, 

7 means for automatically estimating intrinsic colors 

8 of pixels in the area of interest based on color information 

9 for pixels in the first region and the second region after 

10 the user has defined the area of interest. 

1 60. A system for use in extracting a foreground 

2 region from a background region of an image, comprising 

3 means for enabling a user to control an original 

4 extraction by manipulating a brush on a display of the 

5 image , 

6 means for enabling a user to control a touch up 

7 extraction following the original extraction, and 

8 means for considering a pixel identified for touch 

9 up extraction only if the pixel was of uncertain color in 

10 the original extraction. 
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